@grdryn I think we need to test

• APB provisioning
• Dashboards working or not

BTW, have you checked what Grafana version the OpenShift Grafana images use?

2 other -stupid- questions:

1. registry.access.redhat.com/openshift3/grafana this repo is accessible w/o any registration/subscription, right?
2. why don't/can't we use docker hub? if I was a community member I would prefer seeing Docker hub images instead of registry.access.redhat.com.

@aliok Yes, this definitely needs some testing! :) Do you have any links or suggestions on how to test? I'm pretty new to all of this!

have you checked what Grafana version the OpenShift Grafana images use?

5.2.3

registry.access.redhat.com/openshift3/grafana this repo is accessible w/o any registration/subscription, right?

Right! 👍

why don't/can't we use docker hub? if I was a community member I would prefer seeing Docker hub images instead of registry.access.redhat.com.

Honestly, I just initially went for that because the Prometheus image here is already pulling from there. I also don't seem to be able to find a Grafana image from OpenShift on DockerHub, so maybe they haven't published that yet.

I think I understand what you're saying about community preferences. I see it a little differently: it would maybe be better if we referenced non-RHEL images (i.e. from other open source communities such as CentOS or Fedora), in our community APBs. I don't think it should matter that they're on DockerHub or not, as long as they're publicly accessible (could be from docker hub, quay.io, registry.fedoraproject.org, registry.centos.org, or anywhere else (in that vein, is registry.access.redhat.com really any different, given that the images are publicly accessible?)).

Honestly, I just initially went for that because the Prometheus image here is already pulling from there. I also don't seem to be able to find a Grafana image from OpenShift on DockerHub, so maybe they haven't published that yet
Makes sense :)

I think I understand what you're saying about community preferences. I see it a little differently: it would maybe be better if we referenced non-RHEL images (i.e. from other open source communities such as CentOS or Fedora), in our community APBs. I don't think it should matter that they're on DockerHub or not, as long as they're publicly accessible (could be from docker hub, quay.io, registry.fedoraproject.org, registry.centos.org, or anywhere else (in that vein, is registry.access.redhat.com really any different, given that the images are publicly accessible?)).

Was just asking... Good to hear your thoughts.

Do you have any links or suggestions on how to test? I'm pretty new to all of this!

Here's what I will do to verify your changes:

• Provision metrics APB
• Provision Keycloak APB: our Prometheus config should start monitoring the Keycloak instance immediately. Grafana dashboards (included in our Grafana config in metrics APB) should show Keycloak data. In order to see business metrics (# of logins etc) along with resource metrics (CPU usage, etc.) you would need to login to Keycloak and see the numbers changing in Grafana dashboard.
• Create a mobile client and create a binding with metrics service. Then copy the metrics JSON file into the sample app (https://github.com/aerogear/cordova-showcase-template) and run it. App and device metrics in the Grafana dashboards should show the app init events. Run security checks; security dashboard should be updated.

Feel free to do the same tests or let me do it for you.

started verifying this now...

@aliok awesome, thanks! Let me know how you get on, I haven't managed to try this out at all yet (I know...that's a bad thing)!

@grdryn having trouble with apb on OKD 3.11.
Actually, having trouble with apb in general. It doesn't work with OKD 3.9 as well...

@psturc tried apb recently?

Reported ansibleplaybookbundle/ansible-playbook-bundle#321

@grdryn @aliok
No version of APB CLI tool worked well with OKD v3.11 for me.
So as @wei-lee suggested, I've tried to test the APB like this:

1. Clone the PR, build & push the APB to personal dockerhub repo: docker build -t docker.io/aerogeartestcatalog/metrics-apb . && docker push docker.io/aerogeartestcatalog/metrics-apb
2. Edit ASB configmap to pull APB images from my personal repo:

registry:
- black_list:
  - .*-app-apb$
  name: ag
  org: aerogeartestcatalog
  tag: latest
  type: dockerhub
  url: https://registry.hub.docker.com 
  white_list:
  - .*-apb$

3. Redeploy ASB, wait until it's running and refresh Service catalog to check if a new image is there.

After these steps, I tried to provision metrics-apb from Service catalog.
It looks like Prometheus image is working fine, however Grafana pod was crashing and this is the log from its pod:

t=2018-11-20T10:05:25+0000 lvl=info msg="Starting Grafana" logger=server version=5.2.3 commit=unknown-dev compiled=2018-09-27T03:28:27+0000
t=2018-11-20T10:05:25+0000 lvl=info msg="Config loaded from" logger=settings file=/go/src/github.com/grafana/grafana/conf/defaults.ini
t=2018-11-20T10:05:25+0000 lvl=info msg="Path Home" logger=settings path=/go/src/github.com/grafana/grafana
t=2018-11-20T10:05:25+0000 lvl=info msg="Path Data" logger=settings path=/go/src/github.com/grafana/grafana/data
t=2018-11-20T10:05:25+0000 lvl=info msg="Path Logs" logger=settings path=/go/src/github.com/grafana/grafana/data/log
t=2018-11-20T10:05:25+0000 lvl=info msg="Path Plugins" logger=settings path=/go/src/github.com/grafana/grafana/data/plugins
t=2018-11-20T10:05:25+0000 lvl=info msg="Path Provisioning" logger=settings path=/go/src/github.com/grafana/grafana/conf/provisioning
t=2018-11-20T10:05:25+0000 lvl=info msg="App mode production" logger=settings
t=2018-11-20T10:05:25+0000 lvl=info msg="Initializing SqlStore" logger=server
t=2018-11-20T10:05:25+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2018-11-20T10:05:25+0000 lvl=info msg="Starting DB migration" logger=migrator
t=2018-11-20T10:05:25+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Migration failed err: unable to open database file"

@psturc did you have the old services running? have you tried provision a new service?

thanks for the suggestions guys. I will check it out myself too

@wei-lee I used new minishift instance.
But just checked now that redeploying Grafana (using old APB image) with image registry.access.redhat.com/openshift3/grafana:v3.11 doesn't work either.

Thanks for trying this out folks. Do you have any idea what might be going wrong? Do we maybe need to launch the container in a different way or something?

I'll try to get a local environment configured (since I still haven't done even that), and see if I can understand anything about it.

Thanks for trying this out folks. Do you have any idea what might be going wrong? Do we maybe need to launch the container in a different way or something?

Checking it now @grdryn

@grdryn I tried to deploy the image "registry.access.redhat.com/openshift3/grafana:v3.11" manually in a separate project.

I got exactly the same error.

I think this is the source for the that image: https://github.com/mrsiano/openshift-grafana but I am not 100% sure.
Do you know how we can get the source for the image?

@grdryn I guess we first need to ask/solve why deploying registry.access.redhat.com/openshift3/grafana:v3.11 in a clean project fails. maybe you can ask the people in https://github.com/mrsiano/openshift-grafana ?

@aliok I'm not sure of the exact upstream repo, but I'd have thought it was this one: https://github.com/openshift/grafana/

I'm trying to get a local environment so that I can try to figure out what's going on (so that I can at least have some understanding before going to others).

Thanks for the feedback on it, I'll try to have more confidence in future changes before I make PRs! 👍

@aliok I'm not sure of the exact upstream repo, but I'd have thought it was this one: https://github.com/openshift/grafana/

Nice, that should be it. I missed it somehow :)

Final note for now:

[aliok@localhost tmp]$ docker run registry.access.redhat.com/openshift3/grafana:v3.11
t=2018-11-20T14:50:09+0000 lvl=info msg="Starting Grafana" logger=server version=5.2.3 commit=unknown-dev compiled=2018-09-27T03:28:27+0000
t=2018-11-20T14:50:09+0000 lvl=info msg="Config loaded from" logger=settings file=/go/src/github.com/grafana/grafana/conf/defaults.ini
t=2018-11-20T14:50:09+0000 lvl=info msg="Path Home" logger=settings path=/go/src/github.com/grafana/grafana
t=2018-11-20T14:50:09+0000 lvl=info msg="Path Data" logger=settings path=/go/src/github.com/grafana/grafana/data
t=2018-11-20T14:50:09+0000 lvl=info msg="Path Logs" logger=settings path=/go/src/github.com/grafana/grafana/data/log
t=2018-11-20T14:50:09+0000 lvl=info msg="Path Plugins" logger=settings path=/go/src/github.com/grafana/grafana/data/plugins
t=2018-11-20T14:50:09+0000 lvl=info msg="Path Provisioning" logger=settings path=/go/src/github.com/grafana/grafana/conf/provisioning
t=2018-11-20T14:50:09+0000 lvl=info msg="App mode production" logger=settings
t=2018-11-20T14:50:09+0000 lvl=info msg="Initializing SqlStore" logger=server
t=2018-11-20T14:50:09+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2018-11-20T14:50:09+0000 lvl=info msg="Starting DB migration" logger=migrator
t=2018-11-20T14:50:09+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Migration failed err: unable to open database file"

I've added the missing container arg to pass the config file, and it seems to at least not crash now. Unfortunately, it seems that it doesn't have the extra plugins that we want (and assumed were there (@wei-lee and I saw them on some openshift cluster were this image (we thought) was deployed)) 😭

I also noticed that provisioning from the old APB and then trying to just update the grafana deployment to use the new image, it fails with what appears to be a data migration step. It looked to be similar to grafana/grafana#10830, where the resolution was "Grafana needs write access to the disk/volume where database is located". That sounds like Grafana upgrades might be difficult in future 😟

That also made me think of another problem that's not directly related: if we went for the approach I've been thinking about up to now -- "use the version of these images that matches the version of OpenShift" -- how would we handle OpenShift being upgraded underneath them? Our services wouldn't get a new image (unless maybe they were exposed as image streams in some centralized openshift project that we could reference 'latest' in, I don't see them though). Maybe that's a good thing, since we'd then need to be sure that upgrades would "just work" automatically, when it seems like they currently might not.

From the rest of @aliok's test suggestions, here's how far I got:

Provision metrics APB

✔️

Provision Keycloak APB: our Prometheus config should start monitoring the Keycloak instance immediately. Grafana dashboards (included in our Grafana config in metrics APB) should show Keycloak data.

✔️ (although I don't see any data on logins or login errors...does this mean logins to the admin console?)

In order to see business metrics (# of logins etc) along with resource metrics (CPU usage, etc.) you would need to login to Keycloak and see the numbers changing in Grafana dashboard.

❌

I've a feeling that I may have somehow set up Keycloak wrong. I was hoping for a redirect to OpenShift login, maybe that's not correct? It first brings me to this page:

and if I go to "Administration Console", I just get the following login dialog, where I can login with admin for the user and password (we should probably generate the password, if we're going for this login flow, right?):

Create a mobile client and create a binding with metrics service. Then copy the metrics JSON file into the sample app (https://github.com/aerogear/cordova-showcase-template) and run it. App and device metrics in the Grafana dashboards should show the app init events. Run security checks; security dashboard should be updated.

❌ (well, I couldn't actually do this one, so it's more "error" rather than "failure")

I created a mobile client, and bound metrics service. I tried running locally with npm run ionic:serve, but unfortunately that doesn't seem to be enough to run device trust checks, and I just get the following:

I was going to build and put on my device, but then I remembered that I'm running everything in minishift, and I don't think it's easy (if even possible?) to call back to from a device. I'm also not sure how to set any of this up with oc cluster up locally yet (it took me ages to get this far with minishift 🙈)

@grdryn

heavy_check_mark (although I don't see any data on logins or login errors...does this mean logins to the admin console?)

Now that you tried and didn't see anything, I am not sure. But, admin console is basically another app
protected by Keycloak and if I remember correctly it was generating data.

I created a mobile client, and bound metrics service. I tried running locally with npm run ionic:serve, but unfortunately that doesn't seem to be enough to run device trust checks, and I just get the following:

Yeah, that's right. Security checks can't work on the browser.

I was going to build and put on my device, but then I remembered that I'm running everything in minishift, and I don't think it's easy (if even possible?) to call back to from a device. I'm also not sure how to set any of this up with oc cluster up locally yet (it took me ages to get this far with minishift see_no_evil)

LOL. You can access from a device but it is problematic.
But, emulator should be fine I guess? It could resolve the nip.io subdomain and go to the IP on your machine.oc cluster up at least was working in this case I think in my prev experience.

Maybe we can leave all of above to @psturc for verification. Or, I could do it myself.

But, following is a bigger problem:

Unfortunately, it seems that it doesn't have the extra plugins that we want

Can you tell me your APB image location? Let me give it a try on my machine.

@grdryn

✔️ (although I don't see any data on logins or login errors...does this mean logins to the admin console?)

afaik this counts the number of logins/registrations of users for a specific realm (you must first create one via admin console and enable registrations in realm settings)

Unfortunately, it seems that it doesn't have the extra plugins that we want

Can you tell me your APB image location? Let me give it a try on my machine.

Yeah, let's share the apb and let's give it a try :)

Thanks for the feedback guys! The APB is here (the tag is stage): https://hub.docker.com/r/grdryn/grdryn-metrics-apb/tags/

In any case, even if we do need to keep our own custom image, it would make more sense if it were based on this one than our own fork.

👀

Mobile app & security metrics works well. Only thing that's missing is a plugin for piecharts - maybe that was one of the reasons why we had our own image?

Update
Sync & Keycloak is working fine (including their separate dashboards). These are all services using metrics (afaik).

Only thing that's missing is a plugin for piecharts - maybe that was one of the reasons why we had our own image

Yes. #22

@grdryn can you try to install the pie chart plugin (id grafana-piechart-panel) like this?
http://docs.grafana.org/installation/docker/#installing-plugins-for-grafana

@aliok Yeah, I'm on training this week, but when I get a chance, I'm going to try to install the plugins via the APB, similar to how we add the prometheus jar to keycloak from the APB

I've made the changes here to bundle the latest version (1.3.3) of the piechart plugin (details in the relevant commit message), and I've added steps to the grafana provisioning tasks to put it into the PV after Grafana is up, and then restart grafana (scale down & up).

I've played around with it a bit, and I'm relatively happy with it now. I'm not familiar enough with APB testing to know if/how it should be added to apb test

I haven't added the worldmap plugin that was in our old image, because I don't see it referenced in any of our dashboards. Is it a thing we definitely need?

I've added a new commit here that isn't really related to the original PR purpose, but it's small so I've lumped it in. It removes the dependency on jq, since oc can do equivalent filtering with the jsonpath templating anyway. It appears to work fine, here's the MDC after I've bound metrics to an app:

Note that if trying this part out, I've created AEROGEAR-8214 for issues with the unbind operation.

I haven't added the worldmap plugin that was in our old image, because I don't see it referenced in any of our dashboards. Is it a thing we definitely need?

Don't remember seeing worldmap used anywhere. If you didn't see it used we're fine.

Is this PR ready for verification now @grdryn ?

@grdryn

I've made the changes here to bundle the latest version (1.3.3) of the piechart plugin (details in the relevant commit message), and I've added steps to the grafana provisioning tasks to put it into the PV after Grafana is up, and then restart grafana (scale down & up).

Did the 'official' approach not work?

docker run \
  -d \
  -p 3000:3000 \
  --name=grafana \
  -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource" \
  grafana/grafana

Is this PR ready for verification now @grdryn ?

Yes! :)

Did the 'official' approach not work?

I don't think that would be sufficient for our use case unfortunately, since that would call out to the internet at runtime to download code that we don't have control over, right?

Ok, trying it now

@grdryn

I got this for some reason:

+ [[ provision --extra-vars {"GRAFANA_STORAGE_SIZE":10,"POSTGRES_STORAGE_SIZE":10,"POSTGRES_USER":"user","PROMETHEUS_STORAGE_SIZE":10,"_apb_last_requesting_user":"developer","_apb_plan_id":"default","_apb_service_class_id":"e5fcc2ce29065a260252623e7b493861","_apb_service_instance_id":"955919eb-f930-11e8-a0b5-0242ac110008","cluster":"kubernetes","namespace":"asd"} == *\s\2\i\/\a\s\s\e\m\b\l\e* ]]
--
  | + ACTION=provision
  | + shift
  | + playbooks=/opt/apb/actions
  | + CREDS=/var/tmp/bind-creds
  | + TEST_RESULT=/var/tmp/test-result
  | + whoami
  | + '[' -w /etc/passwd ']'
  | ++ id -u
  | + echo 'apb:x:1000640000:0:apb user:/opt/apb:/sbin/nologin'
  | + set +x
  | + [[ -e /opt/apb/actions/provision.yaml ]]
  | + [[ -e /opt/apb/actions/provision.yml ]]
  | + echo ''\''provision'\'' NOT IMPLEMENTED'
  | 'provision' NOT IMPLEMENTED
  | + exit 8

Have you seen something like this?

Perhaps I made a mistake somewhere when building/pushing to my Docker hub.
Do you have any APB images pushed that I can use to test?

@aliok Thanks for taking a look! Can you tell me what you are running there when you get the error?

The stage tag of this image is what I've built from this PR: https://hub.docker.com/r/grdryn/grdryn-metrics-apb/tags/

Ok tried 2 times. Receive this the APB pod logs:

TASK [provision-metrics-apb : Wait for app-metrics health endpoint to report healthy] ***
--
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (30 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (29 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (28 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (27 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (26 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (25 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (24 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (23 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (22 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (21 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (20 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (19 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (18 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (17 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (16 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (15 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (14 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (13 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (12 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (11 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (10 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (9 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (8 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (7 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (6 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (5 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (4 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (3 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (2 retries left).
  | FAILED - RETRYING: Wait for app-metrics health endpoint to report healthy (1 retries left).
  | fatal: [localhost]: FAILED! => {"attempts": 30, "changed": false, "content": "", "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "https://aerogear-app-metrics-aa3.127.0.0.1.nip.io/healthz "}
  |  
  | PLAY RECAP *********************************************************************
  | localhost                  : ok=50   changed=42   unreachable=0    failed=1

But when I try https://aerogear-app-metrics-aa3.127.0.0.1.nip.io/healthz in my browser I see 200.
I don't think this is related to your changes but I am kind of tired from this verification. Let's ask others to give it a try.
cc @jhellar @darahayes

I don't think this is related to your changes but I am kind of tired from this verification. Let's ask others to give it a try.

Oh hey I never even noticed this PR before! I will try to make some time to review and verify this.

I can't get this to build with the changes I've made to the Makefile and Jenkinsfile, so I'm going to back out them, and the changes I've made to the Dockerfile to remove the deprecation notice of the playbook location. I don't care about any of that stuff for the work that I'm doing, so I'll just create a new issue for it.

This is now ready again. I've rebuilt and pushed as docker.io/grdryn/grdryn-metrics-apb:stage.

I've tested on my local minishift by provisioning from the service catalog, and everything works as expected.

I have also deployed to the community cluster using the following command:

oc run metrics-apb-provision --image-pull-policy=Always --env="POD_NAME=metrics-apb" --env="POD_NAMESPACE=grdryn-aerogear-8083" --image=docker.io/grdryn/grdryn-metrics-apb:stage --restart=Never --attach=true --serviceaccount=apb provision -- --extra-vars '{"namespace": "grdryn-aerogear-8083", "PROMETHEUS_STORAGE_SIZE": "10", "GRAFANA_STORAGE_SIZE": "10", "POSTGRES_STORAGE_SIZE": "10"}'

You can see the provisioned services here: https://comm2.skunkhenry.com:8443/console/project/grdryn-aerogear-8083/overview

You can also see the logs for the completed provision pod here (the broker and catalog weren't used here, so the pod ran in the same namespace): https://comm2.skunkhenry.com:8443/console/project/grdryn-aerogear-8083/browse/pods/metrics-apb-provision?tab=logs

I have verified the apb locally, and everything is working as expected. I can see the piechart plugin in grafana as well.

Thanks a lot for helping with this everyone! :)